Program Evaluation and Impact Assessment

This one-week course provides a thorough introduction to the most widely used methods of program evaluation, also known as impact assessment. This is an applied methods course, with equal emphasis on methods and their applications.

The target audience is practitioners and would-be practitioners of policy evaluation who want to measure the effects of policy interventions and policy changes. We also recommend the course to a broader audience of researchers in social sciences who are interested in measuring the effect of interventions and policy changes.

The knowledge and skills developed in this course are essential to understand and effectively participate in policy evaluations. They are necessary for anyone who works with the results of policy evaluations: those who commission evaluations and want to judge feasibility and methodological soundness of competing proposals, and those who use the results of evaluations and want to know the validity and reliability of their results.

The ideal student has some statistical background and is not afraid of working with data but does not have a firm methodological background in program evaluation or econometrics.

The course provides up-to-date methodological knowledge, together with the corresponding intuition and the related skills in handling software, interpreting results and presenting those results. Students successfully completing this course should be able to understand the methodology of an evaluation study, including its fine details, and they should be able to form a well-grounded judgment about its value. They should be able to judge whether the evaluation was carried out in a methodologically sound way given the circumstances, and they should be able to form an educated opinion on how much credibility one should give to its conclusions.

Students who complete this course should be able to design - or effectively participate in a team that designs - an evaluation study using the methods covered in the course (randomized experiments, matching, regression analysis). They should also be able to perform an impact evaluation analysis - or effectively participate in a team performing such an evaluation - using these methods. They should be able to interpret the results of analyses of experimental, matching-based or regression-based policy evaluations. Finally, the students should also be able to use the methods covered in this course in other contexts.

During the course we shall

(a) Give a thorough technical introduction to the methods but emphasize intuition and practical aspects as well

(b) Show the methods in action by going over published evaluations

(c) Replicate some of the published evaluation results using original data, check their robustness and dig deeper if possible

The methods covered in this course are

(1) Randomized Controlled Trials

intent to treat effects, imperfect compliance,  local average treatment effects, encouragement design, power calculations

(2) Matching

exact matching, matching on the propensity score, (and coarsened exact matching, genetic matching, weighting on the propensity score)

(3) Linear Regression

Ensuring common support, functional form choices; regression versus matching

Topics to be covered:

Day1. Intro. to impact evaluation: Why do we need to evaluate? Identifying the counterfactual. Getting the question right and formulating hypotheses. Causal inference vs. association The potential outcomes framework. An example: The Job Corps program in the U.S. Heterogenous Treatment Effects.  A verage Treatment Effects.

Day 2. Randomized Control Trials (RCT):  Intro to RCT. Why randomization helps? Randomization in Practice. Examples. (The NSW program in the U.S.; PROGRESA in Mexico) Intent to treat effects. Imperfect compliance. Local average treatment effects

Day 3. RCT & Matching: Power Calculations (Optimal Design Plus) RCT regressions. Intro to matching: Unconfoundedness. The Common support. Exact matching.

Day 4. Linear Regressions & Matching: Simple linear regression. Ensuring common support. Functional form choices. Regression versus matching. Matching on the propensity score.

Examples in matching.  (Teacher training in Israel.)

Day 5. Matching: Examples in matching (The New Deal for Lone Parents program in the U.K.  Debate based on the NSW program in the U.S.) New methods in matching: coarsened exact matching, genetic matching, weighting on the propensity score

Basic knowledge of statistics and regression analysis, at an undergraduate level, and some knowledge of Stata or R.